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Specification 



Title of the Invention 



Image Processing Apparatus 



Backgro und of the Invention 

The present invention relates to an image 
processing apparatus and, more particularly, to an image 
processing apparatus for generating high-quality image 
data having all color information for each pixel from 
original image data sensed by a single-CCD color 
electronic camera device or the like. 



(solid-state image sensor) for performing photoelectric 
conversion for many pixels arrayed in a matrix on a 
two-dimensional plane, if any pixel is defective, this 
defective pixel cannot attain a correct output, i.e., 
pixel value. In the use of such an image sensor, the 
pixel value of the defective pixel must be compensated 
using the pixel values of pixels around the defective 
pixel. 



single multi-color filtered image sensor, image data 
having all color information (RGB or a luminance value 
and color difference signals) for each pixel is 
generated by interpolation processing from original 
image data of an almost checkered pattern having only a 
pixel value of one original color (e.g., any one of R, G, 



Generally in an image sensor such as a CCD 



In general, in an image sensing device using a 



and B) for each pixel. If original image data obtained 
by the image sensor contains the pixel value of a 
defective pixel, the influence of the defective pixel 
diffuses to neighboring pixels, degrading the image 
quality. 

Conventionally, to compensate for the pixel 
value of such a defective pixel, it is replaced using 
the pixel value of a pixel of the same color one or 
several pixels before the defective pixel, or the 
average of pixels of the same color before and after the 
defective pixel. Further, the pixel value of the 
defective pixel is adaptively replaced based on various 
conditions using a combination of the pixel values of 
two or more pixels before and after the defective pixel 
(see, e.g., Japanese Patent No. 2,636,287 and USP 
5,805,216 ) . 

These pixel value replacement processes can 
make the defect of original image data obtained by an 
image sensor less conspicuous. However, a single 
multi-color filtered image sensor must execute the 
above-described interpolation processing after 
replacement processing. Interpolation processing after 
replacement diffuses the pixel value of the defective 
pixel replaced using neighboring pixel values to 
neighboring pixels, and high-quality image data cannot 
be obtained. 

According to a method of replacing the pixel 



value of a defective pixel using the pixel value of a 
pixel of the same color positioned immediately before 
the defective pixel, when a defective pixel G 43 exists 
near the boundary between a bright region 101 and a dark 
region 102, as shown in Fig. 21A, the pixel value of the 
pixel G 43 is replaced by that of the pixel G 23 of the 
same color positioned immediately before the pixel G 43 . 
As shown in Fig. 2 IB, the pixel value of the pixel G 43 
which should originally represent a bright pixel is 
replaced by a pixel value which represents a dark pixel. 
The pixel value of the replaced pixel G 43 is used to 
generate a pixel value of G color at a neighboring 
interpolation point in subsequent interpolation 
processing. Thus, false information diffuses in 
accordance with the size of the interpolation region. 

More specifically, if a defective pixel exists 
in an interpolation region used to obtain each color 
information at an interpolation point, predetermined 
color information contains the pixel value of the 
defective pixel, so that the pixel value of the 
defective pixel diffuses to all interpolation points 
using the interpolation region containing the defective 
pixel. In an interpolation region of 3 x 3 pixels 
centered on an interpolation point, a dark region of 3 x 
3 pixels centered on the pixel G 43 appears in the bright 
region 101, as shown in Fig. 21C. High-order 
interpolation processing using a wider interpolation 



region in order to ensure a high image quality further 
widens the influence. 

This phenomenon occurs due to not only pixel 
value replacement processing for compensating for a 
defective pixel of an image sensor, but also 
interpolation processing performed after pixel value 
replacement processing . 
Summary of the Invention 

The present invention has been made to 
overcome the conventional drawbacks, and has as its 
object to provide an image processing apparatus capable 
of performing high-speed processing without performing 
adaptive processing based on complicated conditions in 
attaining new image data by interpolating a pixel value 
obtained by an image sensor, and capable of attaining 
high-quality image data while suppressing the influence 
of a pixel replaced by another pixel value prior to 
interpolation processing. 

To achieve the above object, according to the 
present invention, there is provided an image processing 
apparatus for generating new image data having pixel 
values having all color information for each 
interpolation point set on a two-dimensional plane, from 
original image data made up of many pixels which are 
arrayed in a matrix on the two-dimensional plane and 
each of which has only a pixel value representing a 
predetermined color information level obtained by an 



image sensor having individual color filters , comprising 
a replacement unit for, for a pixel value of a pixel 
that need not be replaced by another pixel value among 
pixel values contained in the original image data, 
5 adding replacement information representing 

non-replacement of the pixel value to the pixel value, 
and outputting the pixel value as replacement 
information -added image data, and for a pixel value of a 
O pixel that needs to be replaced by another pixel value, 

m 

m 10 replacing the pixel value by a predetermined pixel value, 

W 

Ln adding replacement information indicating replacement of 

ffl the pixel value to the replaced pixel value, and 

I" 1 outputting the pixel value as replacement 

;~ information-added image data, and an interpolation unit 

15 for outputting interpolated pixel values having all 
jbf color information by interpolating a pixel value at an 

interpolation point for each color information on the 
basis of a predetermined arithmetic expression from 
pixel values of pixels of the same color falling within 
20 a predetermined interpolation region containing the 
interpolation point among all replacement 

information-added image data output from the replacement 
unit, and when replacement information of any pixel used 
for calculation indicates replacement, using an 
25 arithmetic expression different from the arithmetic 
expression. 
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Brief Description of the Drawings 

Fig. 1 is a block diagram showing an image 
processing apparatus according to the first embodiment 
of the present invention; 

Figs. 2A to 2C are views showing an 
arrangement of a replacement unit; 

Figs. 3A and 3B are views showing another 
arrangement of the replacement unit; 

Fig. 4 is a block diagram showing an 
interpolation processing unit according to the first 
embodiment of the present invention; 

Figs. 5A to 5C are explanatory views showing 
arithmetic processing of the interpolation processing 
unit (normal operation); 

Figs. 6A to 6D are explanatory views showing 
arithmetic processing of the interpolation processing 
unit (when a replaced pixel is included) ; 

Figs. 7A to 7C are explanatory views each 
showing another setting example of an interpolation 
point; 

Fig. 8 is an explanatory view showing an 
example of the image processing result when 
interpolation processing is done after a defective pixel 
is replaced; 

Fig. 9 is an explanatory view showing an 
example of the image processing result when the present 
invention is applied to only interpolation processing; 



# 
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Fig. 10 is an explanatory view showing an 
example of the image processing result when the present 
invention is applied to interpolation processing and 
compensation value calculation processing; 
5 Fig. 11 is a block diagram showing an 

interpolation processing unit according to the second 
embodiment of the present invention; 

Figs. 12A to 12C are explanatory views showing 
Q the operation of a region value calculation unit; 

m 10 Fig. 13 is a block diagram showing an 

jfj arrangement of the region value calculation unit; 

Fig. 14 is a block diagram showing 
arrangements of an interpolation unit and compensation 
value calculation unit; 
15 Fig. 15 is a block diagram showing an 

arrangement of an adding unit ; 

Fig. 16 is a block diagram showing an 
arrangement of an A/B coefficient changing unit; 

Fig. 17 is a block diagram showing an 
20 arrangement of an R/B coefficient changing unit; 

Fig. 18 is a block diagram showing an 
arrangement of an HF coefficient changing unit; 

Fig. 19 is a block diagram showing an 
arrangement of a compensation unit ; 
25 Fig. 20 is a block diagram showing an 

arrangement of a defect information generation unit; and 
Figs. 21A to 21C are explanatory views showing 
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conventional image processing. 
Descri ption of the Preferred Embodiments 

The present invention will be described below 
with reference to the accompanying drawings . 

Fig. 1 shows an image processing apparatus 
according to the first embodiment of the present 
invention. Reference numeral 1 denotes image data 
output from an image sensor such as a CCD, which is 
original image data with each pixel value representing a 
color information level corresponding to a predetermined 
color filter; and 10, an image processing apparatus for 
generating from the original image data 1 new image data 
9 made up of pixel values representing all color 
information levels for each interpolation point set on a 
two-dimensional plane. 

In the image processing apparatus 10, 
reference numeral 2 denotes a replacement unit . Based 
on defect information ID representing the 
presence/absence of a defect of the pixel the 
replacement unit 2 replaces, for the pixel value of a 
defective pixel of the original image data 1, the pixel 
value by another pixel value to output replacement 
information representing replacement as replacement 
information-added image data 3 together with the pixel 
value, and outputs, for the pixel value of a 
nondefective pixel, replacement information indicating 
non-replacement as the replacement information-added 



image data 3 together with the pixel value . 

Reference numeral 4 denotes a buffer unit for 
temporarily storing the replacement information-added 
image data 3 sequentially output from the replacement 
5 unit 2; and 6, an interpolation processing unit for 

reading out replacement information -added image data 5 
stored in the buffer unit 4, generating, by 
interpolation processing, pixel values representing all 
^=3 color information levels for a corresponding 

^ 10 interpolation point from the pixel values of a plurality 

i fi 

01 of pixels falling within a predetermined pixel region, 

CP and when pixel values used for interpolation processing 

SJ a 

e includes a pixel value whose replacement information 

yg represents replacement , performing interpolation 

i L-j 

g»j 15 processing using an arithmetic expression different from 
a normal one, and outputting each obtained pixel value 
as the new image data 9 . 

As shown in Fig. 2A, the replacement unit 2 
comprises a delay unit 21 for delaying the original 

20 image data 1 by two pixels, a signal switching unit 22 

for selectively outputting an output from the delay unit 
21 and the original image data 1 based on the defect 
information ID, and a replacement information adding 
unit 24 for adding to an output 23 from the signal 

25 switching unit 22 replacement information (in this case, 
the defect information ID) indicating 
replacement/non-replacement of the pixel value, and 
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outputting the resultant information as the replacement 
information-added image data 3. 

A general CCD discretely, sequentially outputs 
the pixel values of respective pixels as analog values 
5 in the pixel line direction. These pixel values output 
from the CCD are A/D-converted in advance, and 
sequentially input as original image data 1 of digital 
values to the image processing apparatus 10. When the 

pixel R 33 is defective, as shown in Fig. 2B, the pixel 

V ^ 
10 ^ value of the pixel of the same color positioned 

immediately before the pixel R 33 , and held in the delay 

unit 21 is used as the pixel value of the pixel R 33 . If 

the pixel R 33 is not defective, the input original image 

data 1 is directly output . 

15 In general, the defective pixel position of 

the CCD is provided as address information indicating 
the defect position. In this case, a defect information 
generation unit (see Fig. 20) or the like compares an 
address representing the pixel position of the input 

20 original image data 1 with address information of the 
defect position, and outputs the defect information ID 
representing whether to replace the pixel value of the 
original image data 1 (defective pixel), in synchronism 
with the original image data 1 when the addresses are 

2 5 matched. As shown in Fig. 2C, the replacement 
information adding unit 24 adds a bit Dd 
(non-replacement = 0, replacement = 1) representing 
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replacement information to bits DO to Dk of the output 
23 representing a pixel value after replacement 
processing by the signal switching unit 2 2 in accordance 
with the defect information ID, and outputs the 
5 replacement information -added image data 3. 

As the pixel value replacement method, as 
shown in Figs. 3A and 3B, the pixel value of a defective 
pixel may be replaced using the pixel values of pixels 
of the same color positioned before and after the 

G 

l q 10 defective pixel. In this case, as shown in Fig. 3A, two 

fJ; delay units 21A and 21B are series-connected. For a 

J^j nondefective pixel, an output from the delay unit 21A is 

selected; and for a defective pixel, the average of an 
1_ output from the delay unit 2 IB and the original image 

15 data 1 that is calculated by an average calculation unit 
m 24A is selected. If the pixel R 33 is defective, as shown 

Q ^ in Fig. $3B, the average pixel value of a preceding 

pixel R 13 of the same color and a subsequent pixel R 53 of 
the same color is output as a new pixel value of the 
20 pixel R 33 . 

In the above description, the replacement unit 
2 is constituted using a general pixel replacement 
method. However, the present invention is not limited 
to this, and may use another pixel replacement method. 
25 A case wherein the pixel value of a defective pixel is 
to be replaced will be exemplified. The target to be 
replaced is not limited to this, and the present 





invention can be applied to any arrangement so long as 



interpolation processing is done after pixel value 



replacement proces s ing . 




In the present invention, replacement 



5 \±nf ormation indicating replacement/non-replacement of 

the pixel value must be added to each pixel value. This 
replacement information can be added to any position. 
The replacement information may be added to the most 
significant bit side of the pixel value 23, as shown in 
10 Fig. 2C, or to the least significant bit side or between 
bits . 



to the first embodiment of the present invention will be 
described with reference to Fig. 4. Fig. 4 shows an 
15 arrangement of the interpolation processing unit. An 
interpolation processing unit having a function of 
compensating for an interpolated pixel value by a pixel 
value as the representative of the luminance value will 
be exemplified. 

20 In Fig. 4, the interpolation processing unit 6 

is constituted by an interpolation unit 61 for 
performing interpolation processing for the replacement 
information-added image data 5 read out from the buffer 
unit 4, and outputting an interpolated pixel value 62, a 

25 compensation value calculation unit 63 for calculating a 
luminance compensation value 6 4 from a pixel value as 
the representative of the luminance value in the 



The interpolation processing unit 6 according 



- 12 - 



replacement information-added image data 5, and a 
compensation unit 6 5 for compensating for the 
interpolated pixel value 6 2 using the luminance 
compensation value 64, and outputting desired image data 
9. 

The interpolation unit 61 and compensation 
value calculation unit 63 determine whether a replaced 
pixel exists in pixel regions used for their arithmetic 
processes, on the basis of replacement information of 
the read replacement information- added image data 5, and 
select arithmetic expressions used for the arithmetic 
processes in accordance with the determination results. 

Figs. 5A to 5C show the operation of the 
interpolation processing unit 6. Fig. 5C shows normal 
arithmetic expressions used when no replaced pixel 
exists in pixel regions used for interpolation 
processing and compensation value calculation processing 
In the following description, the interpolation point is 
set at a position shifted from each pixel position, i.e. 
between pixels of the original image data 1 (replacement 
information -added image data 5). However, the 
interpolation point is not limited to this , and may be 
set on a pixel. 

In Fig. 5A, an interpolation point Bb is set 
at a position surrounded by pixels R 33 , G 32 , G 43 , and B 42 . 
In this case, the interpolation unit 61 calculates, 
based on equations shown in Fig. 5C, interpolated pixel 



values 62 (gBb, rBb, bBb) from neighboring pixels of the 
same color included in an interpolation region 51G of 2 
x 2 pixels or an interpolation region 51R or 51B of 3 x 
3 pixels containing the interpolation point Bb. 

At the same time, the compensation value 
calculation unit 63 generates a luminance compensation 
value 64 (HFBb) for compensating for the pixel value of 
each color information at the interpolation point Bb in 
accordance with an equation shown in Fig. 5C using the 
pixel values of a plurality of pixels which are 
positioned around the interpolation point Bb and serve 
as the representative of the luminance value, filter 
coefficients shown in Fig. 5B, and a compensation scale 
factor (weighting factor) gf . Calculation of the 
luminance compensation value 64 uses predetermined 
pixels falling within a range centered on the 
interpolation point Bb, e.g., compensation region 52 of 
4x4 pixels corresponding to filter coefficients that 
is wider than a range of pixels around the interpolation 
point used for interpolation processing by the 
interpolation unit 61. 

Each interpolated pixel value 62 calculated by 
the interpolation unit 61 does not contain any high 
spatial frequency components of pixels around the 
interpolation point. To the contrary, the pixel 
compensation value 64 contains a high spatial frequency 
component with a characteristic corresponding to the 



compensation region 52 and the filter coefficients. As 
given by the arithmetic expressions of Fig. 5C, the 
compensation unit 65 adds (or integrates) the pixel 
compensation value 64 to the interpolated pixel values 
5 62 to compensate for the interpolated pixel values 63 
for the respective pieces of color information, and 
calculates new pixel values (g'Bb, r'Bb, and b'Bb) of 
the respective pieces of color information, i.e., 
desired new image data 9 at the interpolation point Bb . 
J: 10 If a replaced pixel exists in pixel regions 

in 

rJ used for interpolation processing and compensation value 

y * 

calculation processing, another arithmetic expressions 
or coefficients set are used as corresponding arithmetic 
Q expressions in accordance with the positional 

Q 15 relationship between the replaced pixel and the 

yt 

p interpolation point, as shown in Figs. 6A to 6D. For 

□ 

example, if a G pixel (pixel G 32 or G 43 ) in the 
interpolation region 51G is a replaced pixel, as shown 
in Fig. 6A, the weight of the replaced pixel is set to 0, 

20 and its pixel value is not used in calculating the G 

component gBb at the interpolation point Bb. The pixel 
value of the other nondefective pixel other than the 
replaced pixel is directly used as the G component gBb. 
If an R pixel (pixel R 33 ) near the 

25 interpolation point Bb in the interpolation region 51R 
is a replaced pixel, the weight of the pixel R 33 is 
reduced in calculating the R component rBb at the 



- 15 - 



interpolation point Bb in order to reduce the influence 
of the replaced pixel. The weights of, e.g., pixels R 33 , 
R 3i' R si' and R 53 in the interpolation region 51R are made 
equal, and the average of these pixel values is used as 
the R component rBb. As shown in Fig. 5A, the 
interpolation regions 51R and 51B are at symmetrical 
positions about the interpolation point Bb. Thus, by 
replacing R pixels in Fig. 6B by B pixels, this 
processing can also be applied to calculation of the B 
component bBb when the pixel B 42 is a replaced pixel . 

If one or more R pixels (pixels R 31 , R 51 , and 
R 53 ) other than an R pixel nearest to the interpolation 
point Bb in the interpolation region 51R are replaced 
pixels, the weights of all the pixels R 31 , R 51 , and R 53 
are set to 0 in calculating the R component rBb at the 
interpolation point Bb in order to reduce the influence 
of the replaced pixels and prevent offset of 
interpolation using the remaining pixels. Then, the 
pixel value of the pixel R 33 nearest to the interpolation 
point Bb is used as the R component rBb . As shown in 
Fig. 5A, the interpolation regions 51R and 51B are at 
symmetrical positions about the interpolation point Bb. 
Hence, by replacing R pixels in Fig. 6C by B pixels, 
this processing can also be applied to calculation of 
the B component bBb when any of the pixels B 22 , B 24 , and 
B 44 are replaced pixels. 

If one or more G pixels (pixels G 21 , G 23 , G 32 , 
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G 34* G 4i' G 43* G 52* and G 54 ) in the compens at ion region 52 
are replaced pixels, as shown in Fig. 6D, the weights of 
the pixels G 21 , G 23 , G 32 , G 34 , G 41 , G 43 , G 52 , and G 54 are set 
to 0 in calculating the pixel compensation value HFBb at 
the interpolation point Bb. As the pixel compensation 
value HFBb, 0 (non-compensation) is used. 

In Figs. 5A to 5C and 6A to 6D, the 
interpolation point Bb is set as shown in Fig. 5A, i.e. , 
a B pixel (pixel B 42 ) exists at the upper right of the 
interpolation point. The above processing can also be 
applied to an interpolation point having another 
positional relationship. When the interpolation point 
is set at a position different from each pixel position, 
i.e., among pixels of the original image data 1 
(replacement information -added image data 5), the 
interpolation point may be set to an interpolation point 
Ab (B pixel exists at the upper left of the 
interpolation point), Ar (R pixel exists at the upper 
left of the interpolation point), and Br (R pixel exists 
at the upper right of the interpolation point) as 
patterns other than Fig. 5A. 

In Fig. 7A, the pixel array example in Fig. 5A 
can be regarded to be horizontally reversed (or rotated 
through 90°). In this case, the interpolation regions 
51R and 51B, and the filter coefficients in Fig. 5B are 
horizontally reversed (or rotated through 90° ), and the 
pixel values of the respective arithmetic expressions 



are exchanged. In Figs. 7B and 7C, R and B pixels in 
Figs, 7A and 5A are regarded to be exchanged. In this 
case, interpolated pixel values are calculated by 
exchanging R and B pixels. In any case, interpolation 
5 processing and pixel compensation processing 

corresponding to the presence/absence of a replaced 
pixel can be executed using the same arithmetic 
expressions in Figs. 5C and 6A to 6D. 

In this manner, when a replaced pixel exists 

Q 

yg 10 in pixels subjected to calculation in the interpolation 

£f| regions 51G, 51R, and 51B used for interpolation 

in 

Si processing and the compensation region 52 used for 

r?j compensation value calculation processing, interpolation 

L_ processing and compensation value calculation processing 

2 15 use arithmetic expressions different from normal ones, 

a | 

in e.g. , arithmetic expressions using reduced weights or a 

Q 

O weight of 0 for the pixel value of the replaced pixel. 

This can reduce or suppress the influence of the 
replaced pixel. Compared to directly using the pixel 
20 value of a replaced pixel replaced by a predetermined 
pixel value obtained from a neighboring pixel , even if 
the replaced pixel and its neighboring pixels have a 
pixel value difference, diffusion of the pixel value 
difference by interpolation processing or pixel 
25 compensation processing after replacement can be 
suppressed to attain a high-quality image signal. 

Figs. 8 to 10 show image processing results 
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representing the influence of a replaced pixel 
(defective pixel). Fig. 8 shows an image processing 
result when interpolation processing is simply performed 
after pixel replacement. Fig. 9 shows an image 
processing result when the present invention is applied 
to only interpolation processing. Fig. 10 shows an 
image processing result when the present invention is 
applied to interpolation processing and compensation 
value calculation processing. In these examples, the 
pixel value of a replaced pixel in a bright region is 
replaced by a pixel value in a dark region near the 
boundary between the dark and bright regions, as shown 
in Figs. 21A to 21C. 

The image processing results in Figs. 8 to 10 
are obtained for an interpolation point set at a 
different position from each pixel position of the 
original image data 1, similar to Fig. 5A. The pixel 
(shown as rectangles) positions of the image processing 
results is not exactly same as the original position 
(not shown) of the replaced pixel. 

In Fig. 8, the pixel value of a replaced pixel 
originally present in a bright region is replaced by a 
pixel value in a dark region, and the replaced value is 
directly used to perform interpolation processing. As a 
result, the pixel value in the dark region is diffused 
by interpolation processing, and several pixels become 
dark at positions surrounded by broken lines in Fig. 8, 



i.e., near the replaced pixel in the bright region. 

In Fig. 9, the present invention is applied to 
interpolation processing. When a replaced pixel exists 
in the interpolation region, an arithmetic expression 
5 different from a normal one, e.g., a reduced weight or a 
weight of 0 for the replaced pixel value is used. 

In Fig. 9, even if the pixel value of a 
replaced pixel is replaced by a pixel value in a dark 
region, the pixel value levels of dark pixels are 
10 reduced at positions surrounded by broken lines in 
Ul Fig. 9, i.e., near the replaced pixel in the bright 

region. Diffusion of the pixel value difference by 



01 



rra 

En 

Lnj interpolation processing after pixel replacement is 

p suppressed. 

. a 

fjj 15 In Fig. 10, the present invention is applied 

™ to interpolation processing and pixel compensation 

^ processing. Almost no pixels become dark at positions 

surrounded by broken lines in Fig. 10, i.e., near the 
replaced pixel within the bright region. Diffusion of 
20 the pixel value difference by interpolation processing 
and pixel compensation processing after pixel 
replacement is suppressed, and high-quality image data 
is obtained. 

The interpolation unit 61 calculates each 
25 interpolated pixel value 62 at the interpolation point 
from the pixel values of pixels of the same color 
falling within each of the predetermined interpolation 
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regions 51G, 51R, and 51B containing the interpolation 
point. The compensation value calculation unit 63 
calculates the pixel compensation value 64 at the 
interpolation point from the pixel values of a plurality 
of pixels falling within the compensation region 52 
wider than and including any interpolation region used 
by the interpolation unit 61. The compensation unit 6 5 
compensates for each interpolated pixel value 62 using 
the pixel compensation value 64. A high spatial 
frequency component which cannot be obtained by 
low-order interpolation in the interpolation unit 61 is 
compensated using the pixel compensation value 64, 
thereby obtaining a new pixel value containing the high 
spatial frequency component . 

As a result, a high-quality image having a 
high frequency component can be attained by relatively 
simple processing without performing high-order 
interpolation for all color information using the pixel 
values of pixels in a wide range or performing 
complicated interpolation processing under various 
conditions around the interpolation point. 

The compensation value calculation unit 63 
calculates the pixel compensation value 64 using only 
the pixel values of a plurality of pixels having color 
information which represents the luminance component of 
an image signal, e.g., using only the pixel values of G 
pixels for an image signal of a Bayer array, as shown in 



Fig. 5A. Thus, the compensation unit 65 can compensate 
for only luminance components for the pixel values of 
pixels of each color information without changing color 
balance. In general, a pixel serving as the 
5 representative of the luminance component is large in 
number and has the highest frequency component. A new 
pixel value containing a higher frequency component can 
be obtained, compared to a pixel value interpolated by 

p only pixels of the same color. 

10 An interpolation processing unit 6A according 

fa to the second embodiment of the present invention will 

zJ be described with reference to Fig. 11. Fig. 11 shows 

y s 

yi an arrangement of the interpolation processing unit. 

O In Fig. 4, the interpolation unit 61 and 

O 15 compensation value calculation unit 63 directly receive 

O "the replacement information -added image data 5 from the 

Q 

buffer. In the second embodiment, as shown in Fig. 11, 
a region value calculation unit 60 is adopted to receive 
and pre-process replacement information -added image data 

20 5, and the replacement information-added image data 5 is 
distributed to an interpolation unit 61A and 
compensation value calculation unit 6 3A. 

In Fig. 11, the region value calculation unit 
60 receives the replacement information-added image data 

25 5, and outputs the logical OR of replacement information 
and the sums of the pixel values of pixels belonging to 
respective pixel regions, as region values 60A for the 
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respective pixel regions set in advance on a sub -matrix 
made up of a plurality of pixels including the 
interpolation point as the center. The region values 
60A calculated by the region value calculation unit 60 
5 are parallel-output in synchronism with reception of a 
pixel block. 

Processes executed in the interpolation unit 
61A and compensation value calculation unit 63A are the 
same as in the interpolation unit 61 and compensation 

;F--s 
z s 

€z 10 value calculation unit 63 in Fig. 4 except that the 

81 

SI interpolation unit 61A and compensation value 

Ul 

En calculation unit 63A do not directly receive the 

Cn 

u\ replacement information-added image data 5, but 

□ selectively use the region values 60A parallel-output 

q 15 from the region value calculation unit 60 to 

~ sequentially calculate and output interpolated pixel 



values 62 and a pixel compensation value 64 at the 
interpolation point on the corresponding sub-matrix. 

Figs. 12A to 12C show the operation of the 
20 region value calculation unit. Fig. 12A shows the 
two-dimensional plane image of the replacement 
information-added image data 5, Fig. 12B shows a 
sub-matrix, and Fig. 12C shows regions set on the 
sub-matrix . 

25 As shown in Fig. 12A, the region value 

calculation unit 60 sequentially receives pixel values 
forming the replacement information-added image data 5 
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by a predetermined number of pixel lines (j direction), 
e.g., four pixel lines as the number of pixel lines 
necessary for calculating the pixel compensation value 
64 in parallel with each other in units of single pixel 
columns as pixel blocks 31. 

As shown in Figs. 12B and 12C, a sub-matrix 32 
centered on an interpolation point x is formed from the 
pixel blocks 31 corresponding to four pixel columns as a 
predetermined number of successively received pixel 
columns (i direction), e.g., the number of pixel columns 
necessary for calculating the pixel compensation value 
64. Respective regions A to L are set in advance in the 
sub-matrix 32 using the interpolation point x as a 
center. Then, the sub-matrix 32 ( and . interpolation 
point x) shifts by one pixel in the i direction on the 
two-dimensional plane image in synchronism with 
reception of the pixel block 31 by the region value 
calculation unit 60. Sequential calculation of a new 
pixel value at the interpolation point x in synchronism 
with reception of the pixel block realizes pipeline 
processing . 

For the respective regions A to L set in 
advance using the interpolation point x as a center on 
the sub-matrix 32 constituted in this way, as shown in 
Fig. 12C, the region value calculation unit 60 
calculates the sums, i.e., region values 60A of the 
pixel values of pixels belonging to the respective pixel 



regions. Then, the region value calculation unit 60 
parallel-outputs the region values in synchronism with 
reception of the pixel block 31. The interpolation unit 
61A and compensation value calculation unit 63A 
selectively use the parallel-output region values, and 
sequentially calculate and output interpolated pixel 
values 62 and a pixel compensation value 64 at the 
interpolation point on the corresponding sub-matrix. 

These pixel regions are set based on 
arithmetic expressions used by the interpolation unit 
61A and compensation value calculation unit 63A. 
Fig. 12C shows the pixel regions A to L when 
interpolation processing and pixel compensation value 
calculation processing in the first embodiment are 
employed. 

A case wherein the pixel regions A to L are 
set as pixel regions in advance will be exemplified. 

Fig. 13 shows an arrangement of the region 
value calculation unit . 

In Fig. 13, reference numerals 11 to 14 denote 
shift registers each made up of three series-connected 
1-pixel clock delays 111 to 113, 121 to 123, 131 to 133, 
or 141 to 143. The shift registers 11 to 14 are 
arranged in parallel with each other for pixel values 
Vil to Vi4 of the pixel blocks 31. The "1 -pixel clock 
delay" (to be referred to as a delay hereinafter) is a 
latch circuit for delaying and outputting an input pixel 



value in synchronism with a pixel clock signal in the 
pixel line direction (i direction). 

When four successive pixel blocks 31 are 
sequentially received, the delays of the shift registers 
5 11 to 14 output pixel values at pixel positions on the 
sub-matrix 32. If necessary, a plurality of pixel 
values constituting each pixel region are added by 
adding units 15 to 18 , and the logical OR of replacement 
information is calculated, thereby obtaining each region 
2 10 value. The region value calculation unit 60 calculates 
!tj the region values 60A (A to L) from the received 

|LJ sub-matrix 32, and parallel-outputs them. 

Fig. 15 shows an arrangement of an adding unit, 



En 
in 

3 Of two replacement information-added image data 150 and 



yp 15 151, pixel values 150A and 151A are added by an adder 
01 152, and pieces of replacement information 150D and 151D 

pj are ORed by an OR gate 153. The sum and OR are output 

as a new replacement information-added pixel value 154. 
Fig. 14 shows arrangements of the 
20 interpolation unit and compensation value calculation 
unit . 

In the interpolation unit 61A and compensation 
value calculation unit 63A, reference numerals 201 to 
209 denote selectors each for selecting and outputting 
25 any one of four inputs (Br, Bb, Ar, Ab) based on a 
selection signal AB/rb representing the positional 
relationship between the interpolation point x and the 
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color component of an upper right or left pixel. In 
this case, A and B correspond to regions A and B in 
Fig. 12C, and Bb represents a B region, i.e., a 
positional relationship in which a B pixel exists at the 
5 upper right of the interpolation point x. 

For the positional relationship in which a B 
pixel exists at the upper right of the interpolation 
point x, the selection signal AB/rb is "Bb" . The 
~. selectors 201 to 209 output signals at their inputs Bb 

10 from their outputs Q. 



Lfj 



Adding units 115 to 118 of the compensation 
value calculation unit 63A have the same arrangement as 
that in Fig. 15. 

A/B coefficient changing units 211 and 212 of 

15 the interpolation unit 61A have an arrangement as shown 
in Fig. 16. Each A/B coefficient changing unit switches 
and selects a coefficient for an arithmetic expression 
for calculating a G component gA (gB) at the 
interpolation point x from these region values on the 

20 basis of replacement information contained in two input 
region values A and D (or B and D). The A/B coefficient 
changing unit 211 calculates a G component at the 
interpolation point x for positional relationships Br 
and Bb on the basis of the arithmetic expressions shown 

25 in Figs. 5C and 6A. The A/B coefficient changing unit 
212 calculates a G component at the interpolation point 
x for positional relationships Ar and Ab . 
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R/B coefficient changing units 221 and 222 of 
the interpolation unit 61A have an arrangement as shown 
in Fig. 17. Each R/B coefficient changing unit switches 
and selects a coefficient for an arithmetic expression 
for calculating an R component rx (or B component bx) at 
the interpolation point x from these color component 
values using the arithmetic expressions shown in 
Figs. 5C, 6B, and 6C on the basis of replacement 
information contained in three input color component 
values rl, r2 , and r3 (or bl, b2 , and b3). 

An HF coefficient changing unit 223 of the 
compensation value calculation unit 63A has an 
arrangement as shown in Fig. 18. The HF coefficient 
changing unit 223 switches and selects a coefficient for 
an arithmetic expression for calculating the pixel 
compensation value 64 (HF) at the interpolation point x 
from these color component values using the arithmetic 
expressions shown in Figs. 5C and 6D on the basis of 
replacement information contained in three input G 
component values gx, gl , and g2. In practice, when a 
replaced pixel is included, the HF coefficient changing 
unit 223 selects and outputs a value representing 
non-compensation . 

Each of the 2 -input selectors shown in 
Figs. 14 to 18 selects the "input 0" side as the output 
Q when a selection signal SEL is "0", i.e., a pixel is 
not replaced, and the "input 1" side as the output Q 
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when the selection signal SEL is "1", i.e., a pixel is 
replaced. 

In this fashion, the interpolation unit 61A 
calculates the interpolated pixel values 6 2 (gx, rx, bx) , 
the compensation value calculation unit 63A calculates 
the pixel compensation value 64 (HF), and the 
compensation unit 6 5 compensates for the interpolated 
pixel values 62 by the pixel compensation value 64. 
Fig. 19 shows an arrangement of the compensation unit. 
In Fig. 19 , reference numeral 81 denotes an integration 
unit made up of a plurality of integrators for 
integrating (dividing) the pixel compensation value 64 
by a power value of 2 . The respective integrators are 
parallel-connected. Reference numeral 82 denotes an 
adder for selectively adding at least one of outputs 
from the integrators of the integration unit 81 based on 
the compensation scale factor (weighting factor) gf . 

Reference numeral 84 denotes an adder for 
individually adding an output 83 from the adder 82 to 
the interpolated pixel values 62 (gx, rx, and bx) , and 
outputting the sums as new interpolated pixel values 
(r'x, b'x, and g'x), i.e., desired new image data 9 at 
the interpolation point that are compensated with the 
pixel compensation value 64. By arbitrarily selectively 
inputting the compensation scale factor gf , the 
interpolated pixel values 62 can be compensated by an 
intensity corresponding to gf. Since the integration 



unit 81 is constituted by a plurality of integrators for 
integrating power values of 2, an arbitrary compensation 
scale factor gf can be integrated to the pixel 
compensation value 64 with a simple circuit arrangement. 
If the reference level of the pixel compensation value 
64 changes in accordance with the position of the 
interpolation point, gf can be automatically switched in 
accordance with positional information of the 
interpolation point to adjust the reference level of the 
pixel compensation value 64. 

As described above, the second embodiment 
adopts the region value calculation unit 60. The region 
value calculation unit 60 calculates, as the region 
values 60A for respective pixel regions set in advance 
on the sub-matrix 32, the sums of the pixel values of 
pixels belonging to the respective pixel regions, and 
parallel -outputs the region values 60A in synchronism 
with reception of the pixel block 31. The interpolation 
unit 61A and compensation value calculation unit 63A 
selectively use the parallel-output region values, and 
sequentially calculate and output interpolated pixel 
values and a pixel compensation value at the 
interpolation point on the corresponding sub-matrix 32. 

The sub-matrix 32 shifts on the 
two-dimensional plane image of the replacement 
information-added image data 5 in synchronism with 
reception of the pixel block 31. At the same time, a 




new interpolated pixel value 9 compensated with the 
pixel compensation value 64 is attained as the 
interpolated pixel value of each color signal at the 
interpolation point corresponding to the sub-matrix 32. 
5 This realizes pipeline processing synchronized with 

reception of the pixel block 31. An interpolated pixel 
value for a higher image quality can be calculated at a 
higher speed, compared to a case wherein interpolation 
processing is done by numerical calculation using DSP or 
10 the like. 

Replacement information -added image data is 
generated by adding replacement information indicating 
replacement/non-replacement to a pixel value replaced by 
a replacement unit 2. The interpolation processing unit 

15 6A reads in the replacement information-added image data, 
and switches by respective circuit units the 
coefficients of arithmetic expressions used for 
interpolation processing and compensation value 
calculation processing based on the replacement 

20 information added to the pixel value. The circuit 

arrangement is greatly simplified in comparison with a 
case wherein replacement information corresponding to a 
pixel value is distributed to each circuit unit 
separately from the pixel value. This is very effective 

25 particularly for pipeline processing in which pixel 

values received in the past are held, delayed, and used 
for calculation because another circuit need not be 
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arranged to individually provide replacement information 
for each pixel value in synchronism with the pixel. 

In the first and second embodiments, a pixel 
region of 4 x 4 pixels is used as a sub-matrix, and the 
interpolation point is set among pixels of the original 
image data 1 (replacement information-added image data 
5). However, the present invention is not limited to 
this. For example, the present invention can also be 
applied to a case wherein the number of pixels on one 
side of the sub-matrix is set to an odd number, and the 
interpolation point is set at the central pixel. 

In Fig. 1, the defect information ID 
indicating the presence/absence of a defect of each 
pixel is input in synchronism with the original image 
data 1. In general, defective pixel position 
information indicating the position of a defective 
pixels among the pixels of a CCD is provided as an 
address on a two-dimensional image output from the CCD. 
The defect information ID must be generated based on 
this information. 

Fig. 20 shows an arrangement of a defect 
information generation unit for generating defect 
information. In this case, defective pixel position 
information of the CCD is expressed by relative address 
information in advance. 

The "relative address information" indicates a 
relative value (difference) between addresses for a 



plurality of defective pixel addresses allocated in the 
pixel scanning direction on a two-dimensional image. 

Each relative address information 91A, 9 IB, 
91C, ... of defective pixels is loaded to an internal 
buffer memory 90 via a data bus 98 by the number of 
defective pixels present on the next pixel line within a 
short period such as a period during which output of 
image data from the CCD stops, e.g., a horizontal 
blanking period. Each of the pieces of relative address 
information 91A, 9 IB, 91C, . . . is made up of a relative 
address value 93 indicating an actual relative address, 
and area information 92 representing the bit area of the 
relative address value. 

An address control unit 94 reads out the 
relative address value 93 of the relative address 
information 91A from the internal buffer memory 90, and 
adds the relative address value 93 to any bit area of a 
defective pixel address register 95 based on the area 
information 92. 

A pixel address is set in a pixel address 
register 96 in synchronism with the original image data 
1. The values of the pixel address register 96 and 
defective pixel address register 95 are compared by a 
comparator 9 7 every pixel clock 4> . 

If these values do not match each other, the 
original image data 1 is determined to be a nondefective 
pixel, and "0" is output as the defect information ID. 




If these values coincide with each other, the original 
image data 1 is determined to be a defective pixel, and 
"1" is output as the defect information ID. Accordingly, 
the defect information ID is obtained in synchronism 
5 with the original image data 1. 

When " 1" representing a defective pixel is 
output as the defect information ID, the internal buffer 
memory 90 selects the next relative address information 
r=b 91B. The relative address information 91B is read out 

£S 10 to the address control unit 94, and the relative address 

m 

fpj value 93 is added and set in the defective pixel address 

rn 

register 95, as described above. Then, the processing 
y ^ waits for the next defective pixel. 

S The address control unit 9 4 checks the 

□ 15 contents of the relative address value 93, and reads in 

Q a plurality of pieces of relative address information 

until a predetermined value (e.g., 0) is kept set. 
Using the pieces of relative address information, a 
relative address value can be expressed. 
20 In general, when an image processing apparatus 

of this type executes high-speed pipeline processing, 
the data bus is often used to input/output various data 
including pixel values. The data bus can be used to 
load the address of a defective pixel position from the 
2 5 memory to the internal buffer memory only within a very 
short time such as a horizontal blanking period during 
which output of pixel data from the CCD stops. 
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Considering this situation, defective pixel 
position information is expressed by relative address 
information, as shown in Fig. 20. If defective pixels 
are positioned close to each other in the address space, 
for example, a plurality of defective pixels are 
generated on a single pixel line, the information amount 
necessary for indicating each defective pixel position 
can be reduced to reduce the storage capacity of the 
internal buffer memory. 

As a larger number of defective pixels 
concentratedly exist on a single pixel line, the time 
required for loading, i.e., time for using the data bus 
becomes shorter than a case wherein defective pixel 
position information expressed by an absolute address is 
loaded. Therefore, even when defective pixel position 
information is loaded within a very short time such as a 
horizontal blanking period, the influence of using the 
data bus during this period on another processing can be 
reduced. High-speed pipeline processing can be realized 
as a whole . 

As has been described above , according to the 
present invention, when a replaced pixel having a 
replaced pixel value is included in pixels in an 
interpolation region used to calculate an interpolated 
pixel value or in a compensation region used to 
calculate a pixel compensation value, the interpolated 
pixel value or pixel compensation value is calculated 



using an arithmetic expression different from a normal 
one. This can reduce or suppress the influence of the 
replaced pixel. Even if the replaced pixel and its 
neighboring pixel have a pixel value difference, 
5 diffusion of the pixel value difference by interpolation 
processing or pixel compensation processing after 
replacement can be suppressed to obtain high-quality 
image data, compared to a case wherein the pixel value 
of a replaced pixel that is replaced by the pixel value 

10 of a neighboring pixel is directly used. Moreover, 

replacement/non-replacement of the pixel value is added 
as replacement information to the pixel value, and 
whether the pixel is a replaced pixel is determined 
based on the replacement information of the pixel value 

15 in calculating an interpolated pixel value and pixel 
compensation value. Thus, high-speed pipeline 
processing can be realized with a relatively simple 
circuit arrangement . 
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